feat: Qwen3-TTS 0.6B voice clone - #14
Merged
Merged
Conversation
Plan for adding Qwen3-TTS 0.6B Base voice clone (ref audio + ref text) as a sibling backend to the existing 1.7B VoiceDesign pipeline. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Local Claude Code state (settings.local.json, worktrees/) shouldn't be tracked. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
API shape, encoder precision, and HF repo policy moved from open questions to confirmed decisions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds two new export scripts plus Makefile targets for the 0.6B Base voice clone variant: - export_speaker_encoder.py: ECAPA-TDNN, mel (B,T,128) -> embed (B,1024) - export_tokenizer_encoder.py: Mimi encoder, waveform (B,T) -> codes (B,16,F) Talker / code predictor / vocoder / embeddings reuse the existing config-driven scripts; the new clone-* Makefile targets pin them to the 0.6B model id and a separate output directory so 1.7B and 0.6B builds don't overlap. Encoders ship FP32 only (locked in plan); quantize_int4 is unchanged. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove help-clone; all clone-* targets already appear in make help via the grep pattern. Add CLONE_MODEL_ID/CLONE_OUTPUT_DIR to the variables section. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Mimi encoder's conv padding calls .item() which creates data-dependent guards incompatible with torch.export/dynamo. Switch to the legacy JIT tracer with a fixed canonical size (10s @ 24kHz = 240k samples). Host code zero-pads reference audio to 240k samples before inference and trims output codes based on the original sample count. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Mimi's encoder_transformer uses create_causal_mask with torch.vmap, which is incompatible with JIT tracing (same issue as the talker). Apply the existing mask_patch before tracing. Also add [N/6] step labels to clone-export for clearer progress. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The Mimi model imports create_causal_mask directly into its local namespace (`from transformers.masking_utils import create_causal_mask`), so the existing module-level patch doesn't reach it. Add an explicit patch of `transformers.models.mimi.modeling_mimi.create_causal_mask`. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
generate_clone_onnx.py chains all 6 ONNX models to produce cloned speech from a reference audio + transcript. No PyTorch at inference time. Pipeline: ref audio → tokenizer encoder (codes) + speaker encoder (mel → embed) → ICL prefill → talker decode loop → code predictor → vocoder → trim ref portion → output WAV. Also adds `make clone-generate` target. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Generate a 7.4s reference clip via macOS TTS (say) at 24kHz mono. Stored at tools/qwen3-tts-onnx/fixtures/ref_clone.wav with gitignore exceptions in both root and tools-level .gitignore files. Makefile clone-generate now defaults to this fixture. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
make clone-fixture uses macOS say + librosa to regenerate fixtures/ref_clone.wav from CLONE_REF_TEXT. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace macOS say-generated fixture with one produced by our own 1.7B VoiceDesign backend (Apache-2.0 licensed Qwen3-TTS output). The clone-fixture Makefile target now invokes cargo run --example synthesize. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use "Speak in a warm and friendly female voice with natural intonation" instead of the flat "Speak naturally and clearly" for a more natural sounding reference clip. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Reference fixture now says "Give every small business the voice of a big one." — the WaveKat tagline, consistent with all other examples. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Regenerate ref_clone.wav with two sentences (6.95s, well above the 3s minimum for voice cloning) using FP32 precision for best quality. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Match how the committed fixture was actually generated. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds sibling Qwen3TtsClone struct with full ONNX pipeline: mel spectrogram, speaker encoder, tokenizer encoder, ICL prefill, talker decode loop, code predictor, and vocoder with ref-code trim. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The speaker_encoder.onnx and tokenizer_encoder.onnx use external .data files for weights, but these were missing from the download manifest. Also apply prepare_onnx_dir to the model root so ORT can resolve HF Hub symlinks for the encoder sessions. - Add .onnx.data entries to CLONE_SHARED_FILES in download.rs - Skip directories in prepare_onnx_dir (safe for model root) - Load encoders from symlink-resolved root directory - Update voice clone doc to reflect implemented status - Add fixtures/README.md documenting ref_clone.wav Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Load ref.wav before model in clone example to fail early, add ref.wav requirement note, and print output info after synthesis in all examples. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Merged
wavekat-eason
pushed a commit
that referenced
this pull request
Apr 17, 2026
## 🤖 New release * `wavekat-tts`: 0.0.2 -> 0.0.3 (✓ API compatible changes) <details><summary><i><b>Changelog</b></i></summary><p> <blockquote> ## [0.0.3](v0.0.2...v0.0.3) - 2026-04-17 ### Added - Qwen3-TTS 0.6B voice clone ([#14](#14)) ### Fixed - auto-resample ref audio to 24 kHz ([#15](#15)) ### Other - add model weights section ([#12](#12)) </blockquote> </p></details> --- This PR was generated with [release-plz](https://github.com/release-plz/release-plz/). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Qwen3TtsCloneRust backend for 0.6B voice-clone inference (speaker encoder → tokenizer encoder → LLM → code-to-wave)Test plan
make test-qwen3passescargo run --example synthesize_clone --features qwen3-ttsproduces audiobaseandclonevariants🤖 Generated with Claude Code